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Abstract. In this paper we present an algorithm for determining whether a subgroup H 
of a non-connected reductive group G is G-completely reducible. The algorithm consists of 
a series of reductions; at each step, we perform operations involving connected groups, such 
as checking whether a certain subgroup of G° is G°-cr. This essentially reduces the problem 
of determining G-complete reducibility to the connected case. 



1. Introduction 

Let G be a connected reductive linear algebraic group over an algebraically closed field of 
characteristic or characteristic p > 0. Following Serre [13], we say a subgroup H of G is 
G-completely reducible (G-cr) if whenever H is contained in a parabolic subgroup P of G, 
then H is contained in some Levi subgroup of P. The definition extends to non-connected 
reductive G as well: one replaces parabolic and Levi subgroups with so-called Richardson 
parabolic and Richardson Levi subgroups respectively (see [2], [12] and Section 2). 

Even if one is interested mainly in connected reductive groups, one must sometimes con- 
sider non-connected groups. For instance, natural subgroups of a connected group, such as 
normalizers and centralizers, are often non-connected. The notion of G-complete reducibility 
is much better understood in the connected case, e.g., see [1], [9], and [10]. In this paper we 
present an algorithm for determining whether a subgroup if of a non-connected reductive 
group G is G-cr. The algorithm consists of a series of reductions; at each step, we perform 
operations involving connected groups, such as checking whether a certain subgroup of G° 
is G°-cr. This essentially reduces the problem of determining G-complete reducibility to the 
connected case. 

An important special case of the general problem described above is the following. Let H 
be a subgroup of G. We say Hacts on G° by outer automorphisms if for each 1 ^ h e H, 
conjugation by h gives a non-inner automorphism of G°. In this case, we may identify H 
with a subgroup of Out(G°). Now suppose also that G° is simple; then H is cyclic except 
for possibly when G° is of type _D 4 . It is convenient when studying conjugacy classes in G° 
to determine the fixed point set of a non-inner automorphism. See e.g., [11, Lem. 2.9] when 
H is cyclic and semisimple (that is, H is of order coprime to p); note that if H is generated 
by a semisimple element then H is G-cr by Theorem 2.5, as semisimple conjugacy classes 
are closed. On the other hand, if H is cyclic and unipotent (that is, if is a p-group) then H 
can be G-cr or non-G-cr. 

We prove the following result, which gives a criterion for G-complete reducibility of H . It 
is an ingredient in our algorithm. In case H is cyclic, this is a special case of a recent result 
due to Guralnick and Malle, cf. Theorem 3.1. 
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Theorem (Corollary 4.5). Suppose G° is simple and if acts on G° by outer automorphisms. 
Then if is G-completely reducible if and only if Cqo(H) is reductive. 

Our work fits into a study begun in our earlier papers [2], [3]. It was shown in [2, Thm. 3.10] 
that if if is a G-cr subgroup of G and N is a normal subgroup of if then N is also G-cr. 
In [3] we considered a complementary question: if if is a subgroup of G, N is a normal 
subgroup of if and iV is G-cr then under what hypotheses is if also G-cr? We gave an 
example (due to Liebeck) with if of the form M x N, where M and N are both G-cr but 
if is not [3, Ex. 5.3]. We also showed this kind of pathological behaviour does not happen 
when G is connected and p is good for G [3, Thm. 1.3]. Here we study the above question in 
the case when iV is the normal subgroup if n G° of if (see the algorithm in Theorem 5.3). 

2. Preliminaries 

2.1. Notation. Throughout, we work over an algebraically closed field k of characteristic 
p > 0; we let k* denote the multiplicative group of k. Let if be a linear algebraic group. 
By a subgroup of if we mean a closed subgroup. We let Z(H) denote the centre of if and 
if the connected component of if that contains 1. For h G if, we let Int^ denote the 
automorphism of if given by conjugation with h. Frequently, we abbreviate Int/^p) by h ■ g. 
If S is a subset of if and K is a subgroup of if, then Ck{S) denotes the centralizer of S in K 
and Nk{S) the normalizer of S in K. Likewise, if S is a group of algebraic automorphisms 
of if, then we denote the fixed point subgroup of S in if by Ch(S). If if acts on a set X, 
then we also write Ch{x) for the stabilizer of a point x G X in if. 

For the set of cocharacters (one-parameter subgroups) of if we write Y(H); the elements 
of Y(H) are the homomorphisms from k* to H. 

The unipotent radical of H is denoted R U (H); it is the maximal connected normal unipo- 
tent subgroup of H. The algebraic group H is called reductive if R U (H) = {1}; note that we 
do not insist that a reductive group is connected. In particular, H is reductive if it is simple 
as an algebraic group. Here, H is said to be simple if H is connected and all proper normal 
subgroups of H are finite. The algebraic group H is called linearly reductive if all rational 
representations of H are semisimple. 

Throughout the paper G denotes a reductive algebraic group, possibly non-connected. 

Definition 2.1. Let H C G be a subgroup. We say that if acis on G° fry cmier automor- 
phisms if for every 1 ^ h £ if , the automorphism Int^ j^o of G° is non-inner, i.e., is not 
given by conjugation with an element of G°. This is equivalent to the condition that if maps 
bijectively onto its image under the natural map G — > Aut(G°) —> Out(G°). 

2.2. G-Complete Reducibility. In [2, §6], Serre's original notion of G-complete reducibil- 
ity is extended to include the case when G is reductive but not necessarily connected (so 
that G° is a connected reductive group). The crucial ingredient of this extension is the use 
of so-called Richardson-parabolic subgroups (R-parabolic subgroups) of G. We briefly recall 
the main definitions here; for more details on this formalism, see [2, §6]. 

For a cocharacter A G Y(G), the R-parabolic subgroup corresponding to A is defined by 

P\ '■= {9 £ G I lim A(a)oA(a) _1 exists}. 

Here, for a morphism of algebraic varieties : k* — > X, we say that lim a _;.o 0(a) exists 

provided that <fi extends to a morphism <fi : k — y X; in this case we set lim^o 0(a) = 0(0). 
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Then Pa admits a Levi decomposition Pa = R U (P\) x L\, where 

L x = {geG\ lim A(a)^A(a)^ 1 = g} = C G (\(k*)). 

a— >0 

We call L\ an R-Levi subgroup of Pa- For an R-parabolic subgroup P of G, the different 
R-Levi subgroups of P correspond in this way to different choices of A G Y(G) such that 
P — P\] moreover, the R-Levi subgroups of P are all conjugate under the action of R U (P). 
An R-parabolic subgroup P is a parabolic subgroup in the sense that G/P is a complete 
variety; the converse is true when G is connected, but not in general ([12, Rem. 5.3]). 

Remark 2.2. For a subgroup if of G, there is a natural inclusion Y(H) C If A € Y(H), 

and if is reductive, we can therefore associate to A an R-parabolic subgroup of if as well 
as an R-parabolic subgroup of G. To avoid confusion, we reserve the notation Pa for R- 
parabolic subgroups of G, and distinguish the R-parabolic subgroups of if by writing P\(H) 
for A G Y(H). The notation L\(H) has the obvious meaning. Note that P\(H) = P X C\ ff 
and L X (H) = L x n ff for A G F(ff). In particular, P A ° = P A (G°) and L° x = L X (G°). If 
A G ^(ff) then the R-Levi subgroups of P X (H) are the P u (P\ (if ))-conjugates of L X (H); in 
particular, any R-Levi subgroup of P X (H) is of the form f fl if for some R-Levi subgroup L 
of P A . 

For later use, we record the following way to construct R-Levi subgroups. 

Lemma 2.3. Let P be an R-parabolic subgroup of G, and let M be a Levi subgroup of P°. 
Then Np(M) is an R-Levi subgroup of P. 

Proof. We may choose A G Y(G) such that P = P A , P° = P X (G°) and M = L X (G°) = L° x . 
We have the Levi decomposition P = R U (P X ) X L x = R U {P X ) x L x . Since L x C N P (L X ) and 
R U (P X ) fl Np(L° x ) = 1 (as R U (P X ) acts simply transitively on the set of Levi subgroups of 
P A °), we conclude that 7V P (M) = iVp(L A ) = L A . □ 

Definition 2.4. Suppose if is a subgroup of G. We say H is G -completely reducible (G-ct 
for short) if whenever if is contained in an R-parabolic subgroup P of G, then there exists 
an R-Levi subgroup L of P with H C L. 

Since all parabolic subgroups (respectively all Levi subgroups of parabolic subgroups) of 
a connected reductive group are R-parabolic subgroups (respectively R-Levi subgroups of 
R-parabolic subgroups), Definition 2.4 coincides with Serre's original definition for connected 
groups [14]. 

Let if be a subgroup of G and let G ^ GL m be an embedding of algebraic groups. Let 
h G if n be a tuple of generators of the associative subalgebra of Mat m spanned by if (such 
a tuple exists for n sufficiently large). Then h is called a generic tuple of ff, see [5, Def. 
5.4]. We recall the following geometric criterion for G-complete reducibility [5, Thm. 5.8]; it 
provides a link between the theory of G-complete reducibility and the geometric invariant 
theory of reductive groups. 

Theorem 2.5. Let H be a subgroup of G and let h G ff n be a generic tuple for if. Then 
H is G-completely reducible if and only if the orbit G ■ h under simultaneous conjugation is 
closed in G n . In particular, if H = (h) is a cyclic subgroup of G, then if is G-completely 
reducible if and only if the conjugacy class G ■ h CG is closed. 
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The following result has been proved with methods from geometric invariant theory (see 
[5, Def. 5.17]): 

Theorem 2.6. Assume that the subgroup H of G is not G- completely reducible. Then there 
exists an R-parabolic subgroup P of G with the following two properties: 

(i) H is not contained in any R-Levi subgroup of P, 

(ii) N G (H) C P. 

The geometric construction of P in [5, §4] is roughly as follows: there is a class of so-called 
optimal destabilizing cocharacters Q C Y(G) such that if A G Q then P := P\ has properties 
(i) and (ii) as in Theorem 2.6. We call such an R-parabolic subgroup P of G an optimal 
destabilizing R-parabolic subgroup for H . 

2.3. Criteria for G-complete reducibility. In this subsection we study criteria for G- 
complete reducibility in terms of some smaller group. 

We first recall the following results about normal subgroups (cf. [3, Thm. 3.4, Cor. 3.7(h)]). 

Theorem 2.7. Suppose that N C H are subgroups of G and N is normal in H . 

(i) If N is also normal in G, then H is G-completely reducible if and only if H/N is 
G / N -completely reducible. 

(ii) // H/N is linearly reductive, then H is G-completely reducible if and only if N is 
G-completely reducible. 

A homomorphism n : G\ — > Gi is called non- degenerate provided that ker(7r)° is a torus. 
The next result is contained in [2, Lem. 2.12 and §6]: 

Lemma 2.8. Let ir : G\ — >■ G-i be a non- degenerate epimorphism of reductive groups. Let 
H C G\ be a subgroup. Then H is G\-completely reducible if and only if n(H) is G%- 
completely reducible. 

As an immediate consequence, we obtain the following result which allows us to focus on 
the part of G that is effectively acting on G°. Note that C G (G°)° C Z(G°)° is a torus. 

Corollary 2.9. Let H C G be a subgroup. Let 71 : G — > G/Cg{G°) be the natural projection. 
Then H is G-completely reducible if and only if n(H) is ir(G)- completely reducible. 

The following lemma gives two necessary conditions for a subgroup of G to be G-completely 
reducible, both of which can be checked in the connected group G°. 

Lemma 2.10. Let H be a G-completely reducible subgroup of G. Then the following hold: 

(i) H n G° is G° -completely reducible; 

(ii) C G o(H) is G° -completely reducible. 

Proof, (i). This is the content of [2, Lem. 6.10 (ii)]. (ii). Since H is G-cr, so is its centralizer 
Cc{H), by [2, Cor. 3.17 and §6]. Now the result follows from part (i). □ 

Under the assumption that assertion (i) or (ii) of Lemma 2.10 holds, the next two lemmas 
allow us to replace the ambient group G with a potentially smaller subgroup M. 

Lemma 2.11. Let H be a subgroup of G and suppose that H(lG° is G° '-completely reducible. 
Then M = HCgo(H D G°) is reductive. Moreover, H is G-completely reducible if and only 
if it is M-completely reducible. 
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Proof. First note that M is a well-defined subgroup of G, since H normalizes G G o(if D G°). 
As Cqo(H fl G°) is a G°-cr subgroup (Lemma 2.10(H)), it is reductive, by [13, Property 4]. 
The same is true for H D G° by assumption. Hence (if D G°)G G o(if fl G°) is the product of 
two reductive groups and thus is reductive. As this group contains M° as a normal subgroup, 
the group M is reductive as well. 

Now first suppose that H is not G-cr. Let A G i^(G) = Y(G°) be an optimal destabilizing 
cocharacter for H in G. Then P\(G°) contains the subgroup H fl G°, which is G°-cr by 
assumption. Hence after replacing A with an P M (PA)-conjugate, we may assume that A 
centralizes H D G°. This implies that A G Y(M), and H C P A (M) C P x . Since if is not 
contained in an R-Levi subgroup of Pa (cf. Theorem 2.6), it is not contained in an R-Levi 
subgroup of P\(M). We conclude that H is not M-cr. 

Conversely, suppose that H is G-cr. Let A G Y(M) and suppose that H C P A (M). To 
show that H is M-cr we need to show that H is contained in an R-Levi subgroup of P\(M). 
Since H is G-cr, we can find a cocharacter \i G Y(G), such that H C C P^ = P A . In 
particular, /i centralizes HnG°, so that // G F(M). Hence H C L M (M) C P M (M) = P A (M), 
as required. □ 

Remark 2.12. In the situation of Lemma 2.11, the subgroup N = H fl G° of H is normal 
in M. Thus, by Theorem 2.7(i), we deduce that H is G-completely reducible if and only if 
H/(H n G°) is M/(H n G°)-completely reducible. 

Lemma 2.13. Let H be a subgroup ofG and suppose that Cgo(H) is G°- completely reducible. 
Then M = HCgo(Cgo(H)) is reductive. Moreover, H is G-completely reducible if and only 
if it is M-completely reducible. 

Proof. We proceed as in the proof of Lemma 2.11: Again, M is well-defined since H normal- 
izes G G o(G G o(#)). Moreover, M° C (H n G )G G o(G G o(PT)) = G G o(G G o( J ff)) C M, yielding 
M° = (G G o(G G o(if)))°. Since G G o(if) is G°-cr by assumption, as before we may conclude 
that its centralizer is reductive, so that M is reductive. 

Suppose that H is not G-cr. Let A G Y(G) = Y(G°) be an optimal destabilizing cochar- 
acter for H in G. By Theorem 2.6(h), P\ contains G G o(if), which is G°-cr. Thus we may 
again assume that A centralizes G G o(if), so that A G Y(M). As before, we conclude that H 
is not M-cr. 

Conversely, suppose that H is G-cr. Let A G Y(M) such that H C P A (M). As A evaluates 
in M°, it centralizes G G o(if), so that G G o(PT) is contained in Pa. On the other hand, since H 
is G-cr, we may find \i G Y(G) such that P M = P\ and such that fi G Y(Cgo(H)). But then 
P M (G G o(P")) = C G o{H), which forces /i to centralize C G o(H) [2, Lem. 2.4]. So /i G F(M), 
and if C L M (M) C P M (M) = P\(M). As before, this shows that H is M-cr. □ 

Remark 2.14. Let if be a subgroup of G and let 7r: G — >• G' be an isogeny. Then 7r(G G o(if)°) = 
G G /o(7r(if))° (see the proof of [4, Lem. 3.1]), so Cg'o(tt(H ))° is reductive if and only if C G o(H )° 
is. 

We may write the connected reductive group G° in the form 
(2.15) G° = SG X ---G n , 

where S 1 is the radical of G° and G\, . . . ,G n are the simple components of the derived group 
of G°. Any subgroup H of G acts via conjugation on the derived subgroup of G° and hence 
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permutes the simple components. We obtain an induced action of H on the set of indices 
{1, . . . , n}. For 1 < i < n, we use the shorthand 

Gi = S J^J Gj 

for the product of all factors in G° above with the exception of G^. 

Our next lemma allows us to replace G with a collection of reductive groups whose identity 
components are simple. 

Lemma 2.16. Let H be a subgroup of G. For 1 < i < n, let Hi := Nn{Gi) and let 
7Tj : HiG° — > HiG°/Gi be the natural projection. Let I C be a subset meeting 

each H-orbit. Then H is G- completely reducible if and only if iTi(Hi) is 7Ti(HiG )- completely 
reducible for each i 6 /. 

Proof. First note that, by construction, Hi and G° both normalize the group Gi. Hence the 
map 7Tj is well-defined. Since HiG° is reductive, so is its image under 7Tj. 

To prove the forward implication, suppose the assertion fails for some i e J. Up to 
reordering the indices, we may assume that it\(Hi) is not 7ri(PiG°)-cr and that H acts 
transitively on the set {l,...,r} for some r > 1. Let Q be an optimal destabilizing R- 
parabolic subgroup of n\(HiG ) for ni{Hi). To obtain a contradiction, we show that ni{Hi) 
is contained in an R-Levi subgroup of Q. Consider the group Q°. This is a parabolic 
subgroup of 7Ti(HiG°)° = iri(G°) = ni(Gi), hence it is of the form Q° = 7r 1 (P 1 ), where Pi is 
a parabolic subgroup of G\. 

Since Pi contains the centre of G\ and 7Ti(Pi) = Q° is normalized by tti(Hi) C Q, it follows 
that Pi is normalized by H\. Indeed, let h G H\. Then h ■ P\ C G\. On the other hand, 
7Ti(/i • Pi) = ni(h) ■ 7Ti(Pi) = 7Ti(Pi). Since ker(7Ti) = G\, this implies that h ■ Pi C P\G\. 
We conclude that h • Pi C Pi(Gi PI G\) = Pi, where we have used that the last intersection 
is central in G\. 

For 2 < j < r, let hj G H be an element satisfying hj ■ G\ = Gj. Let Pj = hj ■ Pi, which is 
a parabolic subgroup of Gj. Since we have just verified that Hi normalizes Pi, the definition 
of Pj does not depend on the choice of hj that transports G\ to Gj. 

We now consider the parabolic subgroup P = SPi ■ ■ ■ P n of G°, where we take Pj = Gj 
for j > r. By construction, P is normalized by H. Indeed, any h G H fixes S under 
conjugation, and permutes the groups G\, . . . , G n . If h maps Gi to Gj with i, j G {1, . . . , r}, 
then (hhi) ■ G\ = Gj, and hence h ■ Pi = (hhi) ■ Pi = Pj. So h also permutes the groups 
Pi, . . . , P r , and thus normalizes P. 

The group Nq(P) is thus an R-parabolic subgroup of G containing H with Nq(P)° = P 
(see [2, Prop. 6.1]). Since H is G-cr, it is contained in an R-Levi subgroup L of Ng(P), 
hence it normalizes the Levi subgroup L° of P. We may write L° = SLi ■ ■ ■ L n for certain 
Levi subgroups Lj of Pj. Then Hi normalizes L\, since Li = LP n G\. This forces tti(Hi) to 
normalize a Levi subgroup of Q° = tti(Pi). By Lemma 2.3, iii(Hi) is contained in an R-Levi 
subgroup of Q, yielding a contradiction. 

To prove the reverse implication, we again assume after reordering the indices that 1 G / 
and that H permutes the set {1, . . . ,r} transitively for some r > 1. Assume that H is not 
G-cr, and that Q C G is an optimal destabilizing R-parabolic subgroup of G containing H. 
Again we want to deduce that H is contained in an R-Levi subgroup of Q, contradicting our 
assumption. 
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Write Q° = SP\ ■ ■ ■ P n , where the Pj are parabolic subgroups of G{. Since H normalizes 
Q°, Hi normalizes Pi = Q°C\Gi. This means that iii(Hi) is contained in A r 7ri (^ lG o)(7ri(Pi)), 
and the latter is an R-parabolic subgroup of 7r 1 (i/ 1 G ). Since 1 G i, the subgroup ni(Hi) 
is contained in an R-Levi subgroup, say M, of this normalizer. Thus 7r 1 (if 1 ) normalizes the 
Levi subgroup M° of 7Ti(Pi), which is hence of the form 7r 1 (L 1 ) for some Levi subgroup Li 
of Pi. 

Since L\ contains the centre of G\, as in the proof of the forward implication (where we 
have proved that H\ normalizes Pi), we may conclude that H\ normalizes L\. Choosing 
again elements hj E H with hj ■ G± = Gj for 2 < j < r, we obtain well-defined Levi 
subgroups Lj := hj ■ L\ of hj ■ P\ = Pj, where the latter equality follows from Pj = Q° (1 Gj. 
Proceeding similarly for the other if -orbits on {1, ... , n} (each of which contains an element 
of I by assumption), we construct an P-stable Levi subgroup L = SL\ ■ ■ ■ L n of Q°. As 
before, by Lemma 2.3, H is contained in an R-Levi subgroup of Q, which gives the desired 
contradiction. This finishes the proof. □ 

3. Cyclic subgroups 

According to Theorem 2.5, a cyclic subgroup of G is G-completely reducible if and only 
if the conjugacy class of a generator is closed. With this characterization we reformulate a 
recent result of Guralnick and Malle, see [8, Thm. 2.3]: 

Theorem 3.1. Let H be a cyclic subgroup of G. Then H is G-completely reducible if and 
only if Cg°(H) is reductive. 

By Lemma 2.10(h), we deduce the following result. 

Corollary 3.2. Let H be a cyclic subgroup of G. Then C G o(H) is G° -completely reducible 
if and only if it is reductive. 

Combining some of our previous reductions, we obtain the following weaker version of 
Theorem 3.1. This is of independent interest, as our arguments allow us to avoid the case- 
by-case considerations that are needed for the proof of Theorem 3.1. 

Theorem 3.3. Let H C G be a cyclic subgroup of prime order. Then H is G-completely 
reducible if and only if Cgo(H) is reductive. 

Proof. The forward implication is clear, by Lemma 2.10(h). Conversely, assume that Cgo(H) 
is reductive. Since linearly reductive subgroups are G-cr (see [2, Lem. 2.6]), we may assume 
that k has positive characteristic p that coincides with the order of H. 

We first show that Cgo(H) is G°-cr. Suppose this fails, and let P C G° be an optimal 
destabilizing parabolic subgroup for Cg°(H) in G . Then H normalizes P, by Theorem 
2.6(h). Let U be the unipotent radical of P. Then Z{U) has positive dimension and is 
normalized by H and P. Up to passing to a characteristic subgroup (the subgroup of 
elements of order dividing p), we may assume that Z(U) has exponent p. Thus Z(U) has 
the structure of an F p -vector space of infinite dimension with an F p -linear if -action. As H is 
generated by an element h E H of order p, there must be infinitely many fixed points of H on 
Z(U). Indeed, on any if-stable finite dimensional subspace W of Z(U) the automorphism 
induced by h may be brought into Jordan normal form with block sizes bounded by p (the 
Jordan normal form exists as h has only eigenvalue 1 E ¥ p ). As each block contributes at 
least p — 1 fixed points, H has at least (p — 1) [dim W/p] fixed points on W, and we can make 
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dirciFp W arbitrarily large. Taking the identity component of the if -fixed points on Z(U) 
hence yields a non-trivial, connected, normal, unipotent subgroup of Cqo(H), contradicting 
the reductivity assumption. We thus conclude that Cqo(H) is G°-cr. 

By Lemma 2.13, it therefore suffices to show that H is M-cr, where M = HCgo(C g o(H)). 
Let Mi, . . . , M r be the simple components of M°. By definition of M, C Mi {H) C M^ZiM^) 
is finite for each i. This forces Hi = Nn{Mi) = 1 for each i. Indeed, since H has prime order, 
Hi ^ 1 would yield that H normalizes Mi, but due to a result of Steinberg (cf. [16, Thm. 
10.13]), no non-trivial cyclic group can act on a simple group via algebraic automorphisms 
with only finitely many fixed points. By Lemma 2.16, we conclude that H is M-cr, as 
required. □ 

4. Outer automorphisms for D 4 

In this section, let -D4 denote an adjoint simple group of type -D4. Amongst the simple 
groups -D4 has the largest outer automorphism group, in that Ont(Di) = S3, the symmetric 
group on 3 letters. We may identify Out(D^) with the set of graph automorphisms in 
Aut(D^) induced by the symmetries of the Dynkin diagram. However, there are other 
subgroups isomorphic to S3 in Aut(Z^4) that act via outer automorphisms. As this is the 
only situation where outer automorphisms of a simple group arise that is not covered by 
Theorem 3.3, we treat this case separately in this section. 

Let T be a maximal torus of D4 with associated root system $. Let A = {a, (3, 7, 6} be a 
set of simple roots for $, where 5 is the unique simple root that is non-orthogonal to every 
other simple root. Let A = u$ G Y(T) be the fundamental dominant coweight determined 
by (a, A) = (f3, A) = (7, A) = 0, (5, A) = 1. For e G $ we denote by u e : G a — > U e a fixed root 
homomorphism onto the corresponding root subgroup of G. 

Let a G Aut(Di) be the triality graph automorphism determined by requiring that for all 
c G k, 

a(u a (c)) = up{c), 0"(«/?( c )) = Mc); °"( n 7( c )) = u a (c), <t(u s (c)) = u s (c). 

Then Cd 4 (ct) is a simple group of type Gi- In fact, T = CV(cr) is a maximal torus of Cd 4 (ct), 
and a = a\f = (3\f = and ft — 5\f form a pair of simple roots with respect to T, 
with corresponding root groups given by u & {c) = ^(0)^/3(0)^(0), u^(c) = us(c). Since A 

evaluates in T, we may regard it as an element of Y{T); we denote this element by A. We 
have (a, A) = 0, 0, A) = 1. 

We begin with a detailed description of triality in the particular case where the ground 
field has characteristic three, using the results of [6] and [7]. 

Proposition 4.1. Assume that p = 3. In Aut(D4) there are exactly two conjugacy classes 
of cyclic groups of order three generated by outer automorphisms. Let (o~i) , (02) be repre- 
sentatives of the respective classes, and let Mi = Co^i) (i = 1,2). Then we may choose 
the labelling such that the following holds: 

(i) Mi is a simple group of type G2; moreover Aut(-D 4 ) -07, the orbit of o\ under conju- 
gation, is closed in KvX(Di). 

(ii) M 2 is an 8-dimensional group with ^-dimensional unipotent radical and corresponding 
reductive quotient isomorphic to SL2/ the orbit Aut(ZA4) -oi is not closed and contains 
o~i in its closure. 
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(iii) We may take o\ = a. 

(iv) Let u = M Q , +j g +7+ 2<5(l). Then we may take a 2 = au. 

(v) With the choices in (iii) and (iv), we have M 2 = Cm^u) = (U a , U- a ) x R U (P\) ^ P\, 
where P^ denotes P\(Mi). 

Proof. By [6, Cor. 6.5, Thm. 9.1], there are precisely two conjugacy classes of cyclic groups 
of order three generated by outer automorphisms, which are denoted by type I and type 
II, respectively. They are distinguished by the structure of the corresponding fixed point 
groups, where type I yields a group of type G 2 , whereas type II in characteristic 3 gives a 
group with the structure described in (ii) (see [6, §9] together with [7, Thm. 7]). This implies 
the first statements of (i) and (ii), as well as (iii). 

Working in the algebraic group Aut(-D 4 ), using <r(A) = A and (a + (3 + 7 + 25, A) = 2 > 
we compute that 

lim \(a)au\(a)~ 1 = a lim \(a)u\(a)~ 1 = a, 

which proves that a is contained in the closure of the orbit through o~u. By [5, Thm. 3.3], to 
show that a and au are not conjugate it is enough to show that they are not _D 4 -conjugate. 
This non-conjugacy follows from [15, §1, Prop. 3.2], as a and au are given in loc. cit. as class 
representatives for distinct unipotent classes in aD^. Moreover, by [15, §11, Lem. 1.15], the 
orbit through a is closed. This proves (iv) and the remaining assertions of (i) and (ii). 

To prove (v), we first note that R u {Py) consists of the root groups for the roots f3,a + (3, 
2a + (3, 3a + (3 and 3a + 2(3. In particular, the semi-direct product (U a , U^ a ) k R u (P^) 
has dimension 8 and is contained in Cm^u) = C M 1 { u za+2j3^-)) ■ As ^ centralizes ±a, the 
semi-direct product is also contained in Py Since clearly Cj^fu) C M 2 , the assertion (v) 
follows by comparing dimensions. This finishes the proof. □ 

We can now characterize G-complete reducibility in the case where G° = D4 and H maps 
isomorphically onto the full group of outer automorphisms of D±. The following result is the 
analogue of Theorem 3.3. 

Theorem 4.2. Let H be a subgroup of G. Assume that G° = D4 and H = S3 acts by outer 
automorphisms on G°. Then H is G- completely reducible if and only if Cg°{H) is reductive. 

Proof. The forward implication is clear by Lemma 2.10. Conversely, assume that H is not 
G-cr. Let h e H be an element of order 3, so that K = (h) is a normal subgroup of index 
2 in H. By the assumption on H, the map tt : G — > Aut(-D4), g 1— > lnt(g)\co is surjective. 
Since ker(7r) = Cq(G°), ti is an isogeny. Hence n(H) is not Aut(D 4 )-cr, by Lemma 2.8. It 
now follows from Remark 2.14 that we can take G to be Aut(D4). 

First assume that p = 3. Let M\ = Crj 4 (a) and M 2 = Cd 4 {o~u) with notation as in 
Proposition 4.1. Then K is a normal subgroup of order 3 and index 2 in H . Since p = 3 
is coprime to 2, we have by Theorem 2.7(h) that K is not Aut(-D 4 )-cr. This implies (by 
Theorem 2.5) that the orbit Aut(-D 4 ) • h is not closed, whence by Proposition 4.1 there exists 
g G G with ghg^ 1 = au. Replacing H with gHg" 1 , we may assume that h = au. Let s G H 
be an element of order 2 such that h and s generate H. Let r G Aut(D 4 ) be the graph 
automorphism of order 2 determined by s, i.e., the graph automorphism that induces the 
same element as s in Out(D 4 ). Let t = /3 V (— 1) G T C M\. Since r and a fix M 1; both 
elements commute with t and u. Moreover, by construction tut = it -1 . This implies that 
rt has order 2 and (rt)(au)(rt) = raru^ 1 = a~ l u~ l = (au)^ 1 . As rt induces the same 
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element as s in Out(Z?4), we can find x G D4 with s = rtx. We conclude that both pairs of 
elements h = au, s = rtx as well as au, rt generate a group isomorphic to S3. In particular, 
x(au)x- 1 = (rt)(Ttx){au)(Ttxy l (Tt) = (rt)(au)- 1 (rt) = au. Thus x G M 2 C M 1 (cf. 
Proposition 4.1(v)), so that s = rtx normalizes M\. As M\ is simple of type G2, it has no 
outer automorphisms. Therefore we may find s' G M\ with Int(s)|Mi = Ie^s')]^. Since M\ 
is adjoint, s' is of order 2. Now 

(4.3) C G o(H)=C M2 (s'). 

Since s = rtx normalizes M 2 , s normalizes Nm 1 {M 2 ) = P~ x (see Proposition 4.1(v)). Hence 
s' G and Cm 2 ( s ') ^ Up to conjugation in P^ we may thus assume s' G T. 

As T is generated by the images of <S V and /3 V , this reduces the possibilities to s' G 
{« v (— 1), /3 V (— 1), <S V (— 1)/3 V (— 1)}. But then s' centralizes ^ 3a+2/ g; or Up, or t/ a+ ^ respec- 
tively. We deduce that Cm 2 ( s ') H R u (P~ x ) is positive-dimensional in each case. Thus Cm 2 ( s ') 
is not reductive, as required. 

Now let p 7^ 3. Then the subgroup of H of order 3 is linearly reductive, in particular 
it is G-cr and Cg°(K) is reductive. Moreover, the group Cqo(K) is connected being the 
fixed point group under a triality automorphism (cf. [6, §9]). Let M = HCqo(K). By 
[3, Thm. 3.1 (b) (ii)] applied to K C H C M, we deduce that is not M-cr. Since K is 
normal in M, by Theorem 2.7(i), if/X is not M/K-ci. But if/if is cyclic of order 2, so we 
may apply Theorem 3.3 to conclude that C(m/k)°{H/K) is not reductive. By construction, 
(M/K)° = C G o(K) and 

(4.4) C (M/K) o(H/K) = C G o(H). 

This finishes the proof. □ 

Having settled the case of -D4, we can combine Theorems 3.3 and 4.2 to characterize G- 
complete reducibility in case G° is simple and the subgroup H acts by outer automorphisms. 

Corollary 4.5. Let H C G be a subgroup acting on G° by outer automorphisms. Assume 
that G° is simple. Then H is G- completely reducible if and only if Cqo(H) is reductive. 

Proof. We may assume that G° is adjoint (cf. the first paragraph of the proof of Theorem 4.2). 
Since H acts via outer automorphisms, we may identify it as an abstract group with a 
subgroup of Out(G°), the finite group of outer automorphisms of G°. As G° is simple, 
Out(G ) is either simple of prime order or G° is of type D4 and Out(G°) = S3. The result 
now follows from Theorems 3.3 and 4.2. □ 

Remark 4.6. In the situation of Corollary 4.5, we always have dim C^o (if) > 0. This follows 
again from the theorem of Steinberg ([16, Thm. 10.13]) in case H is cyclic. The general case 
follows from the identities for Cqq(H) in (4.3) and (4.4). 



5. The Algorithm 

We return to the general situation where H C G is a subgroup of a possibly non-connected 
reductive group. In this section, we are going to establish an algorithm that reduces the ques- 
tion of whether H is G-cr to the question of whether certain subgroups of certain connected 
reductive groups are G-completely reducible. 
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We start with a proposition that recasts some of our earlier results as operations for a 
potential algorithm. We say that the pair (H, G) is completely reducible provided that H is 
G-completely reducible. 

Proposition 5.1. Let (H,G) be a pair consisting of a reductive group G and a subgroup 
H C G. Then each of the following operations replaces (H,G) with pairs of the same form 
(i.e., consisting of a group and a reductive group containing it as a subgroup): 

(01) Let Ti : G —> G/Cc{G°) be the canonical projection. Replace (H, G) with (tt(H), 7t(G)). 

(02) Let G° = SG\ ■ ■ ■ G n be the decomposition as in (2.15), and let H i; m for 1 < i < n 
be defined as in Lemma 2.16. Replace (H,G) with the pairs 

Mifi), MH.G )), . . . , M# n ), K n (H n G°)). 

(03) If H H G° is G° -completely reducible, replace (H,G) with 

(H/(H n G°), HC G o(H n G°)/(H n G )). 

Moreover, H is G-completely reducible if and only if each of the pairs obtained through one 
of these operations is completely reducible. 

Proof. The results follow from Corollary 2.9, Lemma 2.16 and Remark 2.12. □ 

Remark 5.2. In the situation of (02), suppose we are given a set / as in Lemma 2.16. Then 
it is enough to replace (H,G) with the pairs (^(Hi), TTi(HiG°))i e i in (02). 

We are now in a position to give an algorithm that determines whether H is G-completely 
reducible. 

Theorem 5.3. Let H be a subgroup of G. The following algorithm, starting with the pair 
(H,G), reduces the question of whether H is G-completely reducible in a finite number of 
steps to questions of complete reducibility in connected groups: 

Algorithm. Input: a pair (H',G'), where H' is a subgroup of a reductive group G' . 

Step 1 If G'° is not simple, or Cg>(G'°) ^ 1, apply (02) and then (01) to each of the newly 
obtained pairs. Restart instances of the algorithm for each of the new pairs. Then 
H' is G'-cr if and only if each of these pairs turns out to be completely reducible. 

Step 2 Identify H'/(H' H G°) with a subgroup o/Out(G'°). Let n e {1,2,3,6} be the order 
of H' /(H' n G'°). Ifp does not divide n, H' is G'-cr if and only if H' n G'° is G'°-cr, 
and the algorithm stops. 

Step 3 If H'nG'° is not G'°-cr, the algorithm stops with the conclusion that H' is not G'-cr. 

Step 4 If H'nG' = 1, H' is G'-cr if and only if Cqio(H') is reductive. The algorithm stops. 

Step 5 // 1 ^ H'/(H' n G'°) ? S 3 , let M = H'C G , {H' n G'°)/(H' n G'°). Then H' is G'-cr 
if and only if Cm°{H' /{H' D G' )) is M°-cr. The algorithm stops. 

Step 6 If H'/(H' n G'°) = S 3 , apply (03) and restart the algorithm with the new pair. 

Proof. Step 1 is covered by Proposition 5.1. Moreover, each of the new pairs (H",G") 
produced in this step satisfies G"° simple and Cqh{G"°) = 1, so that the algorithm moves 
on to Step 2 after a possible application of Step 1. 

From Step 2 on, we may assume that G'° is simple and Cg'(G'°) = 1. This allows us to 
identify H'/(H' D G'°) with a subgroup of Out(G'°), and yields the constraints on its order. 
If p does not divide n, the quotient H'/(H' n G'°) is linearly reductive. By Theorem 2.7(h), 
H' is indeed G'-cr if and only if W n G'° is G /0 -cr. 
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From Step 3 on, we may assume in addition that p G {2, 3} and that H is not contained 
in G°. The conclusion of Step 3 is correct by Lemma 2. 10 (i) . 
Step 4 is an application of Corollary 4.5. 

Since we have passed Step 3, we may assume that H' n G'° is G'°-cr. Under the condition 
of Step 5, H'/(H' n G'°) is cyclic of prime order. The conclusion of Step 5 thus follows from 
Remark 2.12 and Theorem 3.3. 

Finally, Step 6 is again covered by Proposition 5.1. Moreover, this step is only applicable 
for G'° simple of type D4. As we may assume H' n G'° 7^ 1 and Z(G'°) = 1, the group 
M = H'C G ,o(H' n G'°)/(H' n G'°) featuring in (03) satisfies dimM < dimG". 

It remains to show that the algorithm terminates. Step 1 may restart finitely many 
instances of the algorithm. In each instance the algorithm terminates in Step 2 - Step 5 
if Step 6 is not reached. If Step 6 is applicable, it replaces G' — which is simple of type 
-D 4 — with a group of smaller dimension. This implies that after Step 1 is applied again, Step 
6 cannot be reached a second time, and the algorithm terminates. □ 

Remark 5.4. (i). It follows from the proof of Theorem 5.3 that Step 1, the only step that 
replaces a pair with several new pairs, need only be done at most twice along a path through 
the algorithm. Also, Step 6 only occurs at most once. 

(ii) . There are some situations where shortcuts may be applied to reduce to a connected 
group. First of all, if H° is not reductive, then H cannot be G-cr. On the other hand, if H 
is cyclic, then we may apply Theorem 3.1 to deduce that H is G-cr if and only if Cgo(H) is 
G°-ct. Finally, if H / (H fl G°) is linearly reductive, we can apply Theorem 2.7(ii) to deduce 
that H is G-cr if and only if H fl G° is G°-cr. However, the proposed algorithm gives a 
systematic approach that deals with all possible cases. 

(iii) . If p = 0, then a subgroup H is G-cr if and only if it is reductive ([14, Prop. 4.2]). Of 
course, H is reductive if and only if H° is reductive, which in turn is equivalent to H° being 
G°-completely reducible. 

6. Examples 

We conclude with some examples of the algorithm outlined in Theorem 5.3. 

Example 6.1. Let p = 3, G = Aut(Z?4). Let a be the triality graph automorphism as in 
Section 4. Let H = (cr)K, where K = Gd 4 (ct) is the fixed point subgroup of type G2. We 
follow through the algorithm to deduce that H is G-cr: 

Step 1 is not applicable, as G° = _D 4 is simple and Gc(G°) = 1. In Step 2 we obtain 
n = 3 = p as the order of (a) = H/{H n G°). Now H n G° = K is G-cr (see Corollary 3.2), 
hence Steps 3 and 4 are not applicable. Step 5 applies and leads us to consider the group 
M = HC Da (K)/K = (a)C Di (K). As K is adjoint, we obtain C M o(a) = 1 and thus this 
group is clearly M°-cr. The algorithm stops with the conclusion that H is G-cr. 

Here we have two commuting G-cr subgroups (a) and K of G and their product is also 
G-cr. This is not always the case: see [3, Ex. 5.1]. 

Example 6.2. Let T be a finite group acting transitively on a finite set /. Let Iq G /. Let 
p : T — > M be a homomorphism to a simple group M such that p(Cr{io)) is not M-cr. We 

set 

G = T x Y[ M, 

iei 
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where T acts on the product by permuting the indices. Clearly, G° = Y\ i M. Let d : M — » G° 
be the diagonal embedding. As T commutes with the image of d, we may form the subgroup 

H = { 7 d(p(7)) her}, 

a finite subgroup of G. We claim that if is not G-cr, and use our algorithm to prove it. 

To apply Step 1, we compute that for i E I, N G (Gi) = Cr{i) x G°. In particular, 
Hi = { 7 d(p( 7 )) I 7 e Cr(i)}. We obtain 

^(ff.G ) = ffiG / JjG,- = C r (i) x M =: G-, 

and correspondingly 

^{H) - { 7P ( 7 ) I 7 G Cr(<)} =: flj- 
Since the action of T on f is transitive, we may by Remark 5.2 replace (if, G) with (H', G'), 
where G' = G' in ,H' = if' 

Now Cqi(G'°) = Cr(io), so Step 1 is again applicable and replaces (H',G r ) with the pair 
(H",M), where 

#" = p(tfr(i )). 

By assumption, if" is not M-cr and hence if is not G-cr, by Step 2. 

As a concrete realisation of this example, take T = PGL 2 (q) for q a sufficiently large power 
of p, I = PGL 2 (q)/B(q) where B is a Borel subgroup of PGL 2 , and consider I as a transitive 
T-set by left translation. Let M = PGL2, and let p : V > M the canonical embedding. If 
we take io = B(q), then Gr(io) = B(q) is not M-cr, for q large enough, as B is not M-cr. 
In this example, we have H C]G° — 1 and Cgo(H) = 1 (as CM(B(q)) = 1 for q sufficiently 
large). In particular, this example shows that if may fail to be G-cr even if Cqo(H) and 
HnG° both are G°-cr. 
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